split large data when using brpc streaming #1947
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Aaaaaaron
force-pushed
the
stream_split_mesg
branch
3 times, most recently
from
04a6930
to
67b6b29
Compare
wwbmmm
requested changes
Oct 11, 2022
| int packet_num = ceil((double)length / (double)trans_unit); | ||
|
|
||
| butil::IOBuf split_data; | ||
| for (int j = 0; j < packet_num; j++) { |
There was a problem hiding this comment.
用while (!data->empty()) 来判断是不是更直接。也不用计算packet_num了。
| data_list[i]->clear(); | ||
| fm.set_has_continuation(has_continuation); | ||
| policy::PackStreamMessage(&out, fm, &split_data); | ||
| WriteToHostSocket(&out); |
There was a problem hiding this comment.
如果has_continuation再Write
否则放到最后统一Write
There was a problem hiding this comment.
这里是指如果消息没有达到 max_stream_data_frame_size, 就还是和之前一样放到最后统一 write 吗
|
另外，建议补充下对应单测 |
|
@wwbmmm 多谢 review, ut 一直想加的, 没有想到合适的测法, 请问你这边有啥建议吗 |
Aaaaaaron
force-pushed
the
stream_split_mesg
branch
from
67b6b29
to
1df2f9a
Compare
可以把max_stream_data_frame_size设置成一个比较小的值，然后发一个比较长的包，看是否有分多次发送（通过Socket的GetOrNewSharedPart()->out_num_messages可以获取write的次数） |
|
想到一个问题，如果还没有收到带结束标记的数据帧（has_continuation=false)，这个stream就关闭了（收到close或者主动关闭），_pending_buf已经保存的iobuf数据没有释放的相关代码，可能存在内存泄漏。 |
|
@Aaaaaaron 这个PR和master有冲突了，能否解决一下。并补充单测，谢谢！ |
|
@wwbmmm 好的 谢谢 不好意思最近比较忙😂 |
|
@Aaaaaaron 请问这个可以合并了么 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
https://github.com/apache/incubator-brpc/blob/master/docs/cn/streaming_rpc.md
"目前的实现还没有自动切割过大的消息，同一个tcp连接上的多个Stream之间可能有Head-of-line blocking问题，请尽量避免过大的单个消息，实现自动切割后我们会告知并更新文档"
尝试解决这个 todo.